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ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  last  part 
the  Brief.  Upo:  <"omplotion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  o.  ..<,ial  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  Human  Factors  Technical  Area  is  concerned  with  the  demands  of  the 
future  battlefields  for  increased  human-machine  complexity  to  acquire,  transmit, 
process,  disseminate,  and  utilize  information.  Research  is  focused  on  the 
interface  problems  and  interactions  within  command,  control  and  intelligence 
centers  and  is  concerned  with  such  areas  as  topographic  products  and  procedures, 
tactical  symbology,  user-oriented  systems,  information  management,  staff 
operations  and  procedures,  and  sensor  systems  integration  and  utilization. 

One  area  of  special  interest  is  the  development  of  procedures  to  support 
and  enhance  the  decision  making  process  within  command,  control  and  intelligence 
centers.  The  current  effort  summarizes  a  framework  which  both  identifies 
the  requirements  for  developing  and  evaluating  decision  aids  and  organizes  de¬ 
cision  aids  according  to  the  functions  they  serve  for  the  decision  maker.  This 
framework  thus  provides  a  concise  way  of  categorizing  already  existing  de¬ 
cision  aids  as  well  as  provides  guidelines  for  developing  and  evaluating  new 
aids.  Examples  drawn  from  the  areas  of  Army  command  and  control  and  intelligence 
are  used  to  demonstrate  the  usefulness  of  the  framework. 

Research  in  decision  aiding  is  conducted  as  an  in-house  effort  with  addi¬ 
tional  support  from  contracting  organizations  which  are  selected  for  their 
unique  contributions  to  this  area.  This  effort  is  responsive  to  the  require¬ 
ments  of  Army  Project  2Q162717A790 , 


A  DECISION  SUPPORT  FR^MFWORK  FOR  DECISION  AID  DESIGNERS 


BRIEF 


Requirement: 

To  develop  and  demonstrate  a  framework  for  the  developers  of  decision 
aids  for  organizing  and  categorizing  decision  aiding  procedures. 


Approach : 

The  development  of  the  framework  was  based  on  a  study  of  existing  de¬ 
cision  aids,  identifying  differences  and  similarities  in  the  purpose,  de¬ 
cision,  execution,  and  evaluation  of  the  aids.  The  largest  differences  were 
in  the  service  provided  to  the  decisionmaker ,  e.g.,  provide  information  or 
logical  reasoning  support. 


Product : 

The  framework  developed  (a)  lists  the  steps  a  decision  aid  designer 
should  ideally  complete  in  the  full  development  and  implementation  of  the 
aid  or  aiding  system,  and  (b)  distinguishes  two  categories  of  aids,  those 
that  provide  information  and  those  that  provide  support  for  logically  and 
rationally  evaluating  and  integrating  information  in  making  a  decision. 

See  Figure  1  for  a  schematic  of  the  framework.  Army  command  and  control  ex¬ 
amples  of  the  two  types  of  decision  aids  are  summarized  to  help  explain  the 
use  of  the  framework  as  well  as  distinguish  the  types  of  decision  aids. 


Utilization: 

This  framework  should  provide  decision  aid  designers  with  an  outline  of 
steps  to  be  followed  in  the  cycle  of  development,  from  initial  conceptuali¬ 
zation  and  implementation  to  evaluation  and  revision  of  the  aid.  Special 
attention  is  devoted  to  the  definition  of  requirements  prior  to  aid  develop¬ 
ment  and  an  evaluation  of  the  completed  aid,  two  areas  that  rarely  are  given 
adequate  weight  in  aid  development.  Use  of  the  framework  by  system  develop¬ 
ers,  both  of  large  automated  systems  and  small  manual  procedures,  could  en¬ 
hance  tne  useability  of  their  systems  and  procedures  since  applying  the 
framework  requires  a  thorough  study  of  the  decisionmaker's  requirements  and 
constraints  as  well  as  identification  and  categorization  of  the  decision 
aids  to  be  developed.  Since  the  framework  is  strongly  based  on  psychologi¬ 
cal  principles,  it  also  provides  an  alternative  to  the  more  popular  systems 
and  engineering  perspectives  on  decision  aiding. 
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A  DECISION  SUPPORT  FRAMEWORK  FOR  DECISION  AID  DESIGNERS 


INTRODUCTION 

The  need  for  decision-making  support  is  being  increasingly  recognised 
in  a  variety  of  disciplines:  military  (Kibler,  Watson,  Kelly,  &  T'helps, 
li'78}  Levit,  Alden,  Erikson,  &  Heaton,  1977),  public  policy  making  (Hammond, 
Ronrbaugh,  Mumpower,  &  Adelman,  1977) ,  land  management  (Gardiner  &  Edwards, 
1975) ,  medicine  (Fryback  &  Thornbury,  1978) ,  and  oil  exploration  (von  Winter- 
feldt,  1976).  Decision  aids  are  currently  under  widespread  development,  but 
littla  effort  has  been  devoted  to  examination  of  the  various  functions  de¬ 
cision  aids  fulfill  or  to  development  of  a  system  or  framework  for  coordi¬ 
nating  related  decision  aids. 

The  purpose  of  this  paper  is  to  present  a  preliminary  Decision  Support 
Framework  which  is  intended  to  be  used  by  the  designers  of  decision  support 
systems  to  organize  and  integrate  different  types  of  decision  aids  into  a 
unifying  system.  The  framework  is  both  a  descriptive  tool  to  clarify  the 
relationship  among  decision  aids  and  a  type  of  aid  itself.  It  will  provide 
the  decision  aid  designer  with  a  systematic  context  in  which  to  develop  aids 
as  well  as  to  examine  which  aspects  of  the  decision  problem  would  most  bene¬ 
fit  from  decision  aidinq. 

Before  presenting  the  decision  support  structure,  however,  a  few  gen¬ 
eral  comments  concerning  the  nature  and  function  of  the  type  of  decision 
support  being  discussed  will  help  to  define  the  scope  and  limitations  of 
this  paper.  The  role  of  decision  support  is  to  increase  the  range  of  a  de¬ 
cisionmaker's  capabilities  to  make  a  rational  decision.  Such  a  function  is 
accomplished  by  providing  a  decisionmaker  with  am  informational  base,  as 
well  as  organizational ,  computation,  and  psychological  tools  for  making  a 
logical  decision  based  on  that  information.  Implicit  in  this  role  are  two 
assumptions:  (a)  Decision  support  is  used  when  human  judgment  is  a  criti¬ 

cal  element,  and  (b)  decision  support  in  no  way  replaces  the  decisionmaker 
as  a  problem  solver.  By  definition,  these  are  tools  to  support  the  human 
judgment  and  decision-making  process. 

The  decision  support  framework  presented  in  this  paper  encompasses 
the  various  functions  decision  aids  may  serve,  the  relationship  among  dif¬ 
ferent  aids,  and  an  evaluation  of  those  aids.  While  the  framework  is  dis¬ 
cussed  in  generic  terms,  clearly  the  "tructure  and  configuration  of  specific 
decision  aids  will  depend  on  the  nature  of  the  decision  problem,  the  type 
of  information  required,  and  the  consequences  of  the  decision.  Conversely, 
while  the  examples  of  different  types  of  decision  aids  will  be  drawn  from 
Army  tactical  intelligence  decisionmaking,  the  classes  of  aids  are  intended 
to  be  general  and  apply  to  a  variety  cf  decision-making  contexts. 

The  remainder  of  the  paper  is  organized  into  three  main  sections: 

(a)  an  overview  of  the  Decision  Support  Framework;  (b)  a  more  detailed 
discussion  of  functions  of  and  relationships  among  specific  classes  of 
decision  aids;  and  (c)  factors  to  be  included  in  an  evaluation  of  the  de¬ 
cision  aids. 


DECISION  SUPPORT  FRAMEWORK 


In  Figure  1  is  a  diagram  of  the  Decision  Support  Framework.  This  frame¬ 
work  is  designed  to  depict  the  decision  support  components ,  not  the  decision 
process  itself.  The  pivotal  point  of  the  framework  is  of  course  the  actual 
DECISION  to  be  made.  Based  on  an  analysis  of  that  decision  the  REQUIREMENTS, 
or  the  information  and  tools  needod,  to  make  the  decision  are  defined.  Such 
requirements  may  include  type  of  information,  timeliness  of  information,  data 
computations  needed,  and  how  the  information  should  be  organized,  integrated, 
and  evaluated.  Lists  and  taxonomies  such  as  those  generated  by  Brown  and 
Ulvila  (1976)  may  be  useful  in  determining  the  REQUIREMENTS  for  the  specific 
decision  problem.  The  importance  of  this  analysis  should  not  be  underempha¬ 
sized  since  it  is  the  basis  for  determining  the  content  and  nature  of  the 
decision  aids.  In  addition,  clear  and  systematic  analysis  of  the  require¬ 
ments  is  critical  for  insuring  the  validity  of  the  EVALUATION  of  the  decision 
aids. 


Depending  on  the  requirements  established,  the  types  of  decision  aids 
needed  are  determined.  At  the  most  basic  level,  there  are  two  types  of  IN¬ 
FORMATION  AIDS.  Data-based  aids  make  available  to  the  decisionmaker  the  data 
on  which  the  decision  is  based;  these  aids  can  bent  be  viewed  as  automated 
data  banks  which  make  selected  raw  data,  or  perhaps  summarized  data,  avail¬ 
able  to  the  decisionmaker.  Data  aids  may  select  the  data  for  a  decisionmaker 
based  on  either  predetermined  or  user-specified  criteria.  Calculation  aids 
provide  the  decisionmaker  with  the  results  of  statistical  computations  or 
other  mathematical  computations  such  as  numerical  changes  in  the  data  bases, 
distances,  velocities,  objective  probabilities .  Regardless  of  its  form,  the 
output  of  both  aids  is  information  or  selected  data. 

In  many,  perhaps  most,  situations  the  decisio;imaker  may  feel  overwhelmed 
with  the  sheer  volume  of  information  available  and  have  great  difficulty  in 
selecting  the  most  relevant  information  and/or  making  a  logical  evaluation  of 
that  information.  INTEGRATION  AIDS  are  sets  of  procedures  designed  to  help  a 
decisionmaker  logically  evaluate  and  integrate  the  information  provided  by 
INFORMATION  aids.  The  composition  and  organization  of  specific  aids  depends 
on  the  characteristics  of  the  decision  problem  as  well  as  the  psychological 
difficulties  known  to  influence  this  class  of  decisions.  Thus,  INTEGRATION 
aids  can  serve  a  single  or  a  combination  of  functions,  such  as  those  listed 
in  Figure  Is  organizing  and  structuring  the  information,  helping  to  overcome 
judgmental  and  cognitive  limitations  and  biases  such  as  faulty  memory,  and 
simplifying  the  evaluation  and  weighting  of  the  information.  The  goal  of 
every  INTEGRATION  aid  is  to  help  the  decisionmaker  arrive  at  a  logical,  ra¬ 
tional  DECISION,  not  to  replace  the  decisionmaker. 

Once  the  DECISION  has  been  made,  an  EVALUATION  of  the  various  aids 
within  the  decision  support  framework  is  necessary  for  the  decision  aid  de¬ 
signer  to  determine  if  in  fact  they  did  ’ielp  the  decisionmaker  reach  a  ra¬ 
tional  decision.  As  shown  in  Figure  1,  such  an  evaluation  should  consider 
at  least  the  validity  and  reliability  of  the  aid  as  well  as  its  flexibili  ty 
and  the  degree  to  which  it  led  to  an  improved  decision. 

In  summary,  the  proposed  Decision  Support  Framework  is  designed  to  help 
logically  define  and  organize  the  decision  problem,  describe  the  classes  of 
necessary  information,  and  provide  a  manageable  structure  for  evaluating 
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various  decision  aids.  The  following  sections  of  the  paper  will  focus  on  a 
more  detailed ’discussion  of  two  types  of  decision  aids,  INFORMATION  and  IN¬ 
TEGRATION,  and  their  EVALUATION. 


DECISION  AIDS 

The  purpose  of  decision  aids  within  the  Decision  Support  Framework  is 
to  provide  the  decisionmaker  with  data  on  which  to  base  a  decision  or  with 
help  in  evaluating  the  data  for  a  specific  oroblem.  While  the  two  classes 
of  aids,  INFORMATION  and  INTEGRATION ,  are  indeed  based  upon  different  pur¬ 
poses,  such  a  distinction  is  not  always  cle  >'  in  practice.  The  present 
framework  defines  the  class  of  aid  with  respect  to  its  function  for  the  de¬ 
cisionmaker.  Thus,  although  an  aid  may  integrate,  summarize,  or  perform 
some  algorithm  on  the  data,  if  the  output  of  the  aid  is  a  piece  of  informa¬ 
tion  or  a  restructuring  of  information  it  i >  considered  an  INFORMATION  aid. 
Only  when  an  aid  helps  the  decisionmaker  perform  the  summarization  or  evalu¬ 
ation  of  the  information  is  it  classified  as  an  INTEGRATION  aid.  Such  a 
distinction  is  justified  by  the  focus  of  the  Decision  Support  Framework  on 
the  actual  decisionmaker  as  the  reference  or  pivotal  point,  rather  than  the 
data  or  other  environmental  demands. 


Informational  Aids 

Clearly  no  rational  decision  can  be  made  without  data  or  information. 
However,  in  many  situations  decisionmakers  feel  they  do  not  have  the  type, 
accuracy,  or  timeliness  of  information  used  to  make  a  rational  decision. 

As  shown  in  Figure  1  two  of  the  many  possible  functions  INFORMATION  aids 
could  serve  are  to  provide  data  and  calculations  on  those  data  to  the  de¬ 
cisionmaker.  However,  since  the  needs  for  INFORMATION  aids  will  totally 
depend  on  the  specifics  of  the  decision  problem,  only  a  few  general  com¬ 
ments  and  an  example  will  be  presented.  DATA  aids  are  designed  to  fulfill 
this  function,  usually  by  automating  a  data  base  such  that  large  amounts  of 
data  can  be  stored  and  readily  retrieved  by  a  decisionmaker.  Data  may  be 
stored  as  individual  items  of  information  or  summarized  over  some  specified 
time  period  or  level  of  detail.  CALCULATION  aids  provide  additional  infor¬ 
mation  by  making  computations  on  the  raw  data  such  as  velocity,  percentages, 
or  even  moi  e  sophisticated  algori  thms  involving  matching  templates  or  statis 
tically  optimal  solutions.  However,  for  both  DATA  and  CALCULATION  aids  the 
output  of  the  aid  is  information  which  then  must  be  evaluated  and  integrated 
by  the  decisionmaker. 


Example;  Information  Aid.  The  U.S.  Army  Research  Institute  has  devel¬ 
oped  a  Graphic  Movement  Analysis  Aid  (GRAMA)  to  provide  an  Army  tactical 
commander's  staff  with  decision  support.  The  specific  decision  problem  is 
to  determine  which  avenue  of  approach  the  enemy  is  most  likely  to  select. 

The  purpose  of  GRAMA  is  to  identify  those  routes  which  minimize  travel  time 
and  therefore  are  most  likely.  Specifically,  GRAMA  will  help  an  intelli¬ 
gence  analyst  answer  questions  such  as;  "How  long  will  it  take  an  enemy 
unit  to  move  from  location  A  to  location  B,  and  what  route  will  they  follow? 
A  similar  question  may  be  asked  concerning  the  optimal  routes  for  as  many  as 
10  units  simultaneously  moving  from  10  locations  to  10  destinations.  To 
perform  the  calculations  necessary  for  these  questions,  the  computer 


algorithms  require  data  on:  (a)  the  road  network — what  is  connected  to  what 
by  what  type  of  road;  (b)  the  conditions  of  the  move — day /night,  wet/dry, 
vehicular/foot  travel;  (c)  the  types  of  units  involved — how  fast  can  they 
move  given  the  conditions  and  the  roads.  GRAMA  has  a  preloaded  network  of 
up  to  600  points  and  their  interconnections,  and  a  "Speed"  table  which  de¬ 
fines  how  fast  an  average  unit  can  move  under  a  combination  of  road  and  en¬ 
vironmental  conditions.  The  main  interactive  features  of  the  program  allow 
the  user  to  work  within  this  structure.  Nodes  may  be  added  or  deleted  from 
the  network,  or  specific  subnetworks  of  interest  within  the  larger  network 
may  be  defined. 

Thus,  the  output  of  GRAMA  is  a  rank  ordering  of  various  avenues  of  ap¬ 
proach  based  on  the  available  data  concerning  environmental  conditions, 
enemy  units,  etc.  The  commander's  staff  must  now  incorporate  this  with 
other  information  in  determining  the  most  likely  enemy  avenue  of  approach. 


Integration  Aids 


The  general  purpose  of  INTEGRATION  aids  is  to  help  a  decisionmaker 
evaluate  information  from  data  aids,  calculation  aids  as  well  as  other 
sources,  and  then  to  integrate  that  information  to  make  a  rational  decision. 
INTEGRATION  aids  are  designed  to  help  a  decisionmaker  ultimately  relate  all 
the  relevant  information  to  the  various  decision  alternatives  under  consid¬ 
eration.  Therefore,  definition  of  decision  alternatives  is  a  critical  pre¬ 
liminary  step  to  the  use  of  INTEGRATION  aids.  While  a  decisionmaker  may 
require  help  and  thus  one  aspect  of  aiding  may  be  devoted  to  alternative 
definition,  it  is  assumed  for  the  present  framework  that  the  alternatives 
have  been  defined  either  in  the  decision  REQUIREMENTS  or  as  a  result  of  the 
INFORMATION  aids. 


There  is  ample  evidence  from  psychological  literature  that  several 
factors  degrade  the  quality  of  decisions  which  are  based  on  even  moderate 
amounts  of  information  from  single  or  multiple  sources.  The  primary  func¬ 
tion  of  INTEGRATION  aids  is  to  help  overcome  or  circumvent  these  factors. 

For  any  one  decision  problem  the  degrading  factors  must  be  identified  and 
the  aids  developed  to  take  into  account  those  factors.  While  identification 
of  the  degrading  factors  requires  considerable  analysis  and  research  on  the 
specific  decision  problem,  knowledge  of  the  factors  already  known  to  inter¬ 
fere  will  help  focus  such  efforts.  Since  an  exhaustive  list  is  beyond  the 
scope  of  this  paper,  reference  should  be  made  to  fairly  complete  descriptions 
available  elsewhere  (Tversky  &  Kahneman,  1974;  Hogarth,  1975;  Lichtenstein, 
Fischhoff,  &  Phillips,  1977;  Slovic,  Fischhoff,  &  Lichtenstein,  1977;  Fisch- 
hoff  &  Slovic,  in  press) .  Three  categories  of  factors  which  influence  a 
decisionmaker's  integration  of  information  are  listed  in  Figure  1  and  briefly 
summarized  here. 


Structure  of  Information.  The  output  of  INFORMATION  aids,  in  addition 
to  other  sources,  will  make  available  to  the  decisionmaker  a  large  volume 
of  data,  summaries  of  data,  or  calculations  on  which  to  base  a  decision. 
Perhaps  the  most  basic  function  an  INTEGRATION  aid  can  perform  is  to  organize 
and  structure  this  potentially  overwhelming  mass  of  information.  Such  struc¬ 
turing  serves  at  least  two  functions:  It  allows  all  information  to  be  cate¬ 
gorized  such  that  none  is  unintentionally  overlooked  and  it  also  encourages 


a  mora  logical  evaluation  of  information  which  varies *in  type  as  well  as 
source.  The  most  popular  structuring  techniques  are  based  on  decomposing 
the  decision  problem  into  a  series  of  distinct  categories  of  factors.  These 
factors  may  be  exclusive  of  each  other,  may  be  hierarchical  in  nature,  i.e., 
some  are  subsets  of  others,  or  some  combination  of  the  two.  Once  such  a 
structure  is  formed,  the  information  can  be  sorted  into  the  defined  cate¬ 
gories.  Alternatively  the  structure  can  be  used  to  determine  which  infor¬ 
mation  needs  to  be  gathered  to  arrive  at  a  logical  solution. 

Weighting  of  Information.  One  of  the  most  difficult  tasks  after  the 
information  has  been  organized  into  categories  or  factors  is  to  assess  which 
information  is  relevant  for  making  a  particular  decision.  The  basis  for  de¬ 
termining  relevance  is  diagnosticity ;  that  is,  the  information  that  is  most 
relevant  is  that  which  differentiates  or  discriminates  among  the  options 
being  considered.  For  example,  a  decisionmaker  is  selecting  a  new  car  and 
the  information  obtained  about  the  potential  cars  is  categorized  into  two 
factors:  gas  mileage  and  cost.  If  the  three  cars  A,  B,  and  C  all  get  30 
miles  per  gallon,  then  gas  mileage  is  not  diagnostic  because  it  does  not 
differentiate  the  three  cars  under  consideration.  However,  if  the  prices 
are  A  =  $3,000,  B  =  $4,000,  C  =  $5,000,  then  price  is  a  highly  diagnostic 
factor  since  it  does  differentiate  the  options.  The  psychological  concept 
of  diagnosticity  is  ofr.en  very  difficult  for  decisionmakers  to  consistently 
apply.  In  the  above  example,  it  may  be  hard  for  a  decisionmaker  who  is  en¬ 
ergy  conscious  to  accept  that  gas  mileage  is  not  relevant  when  deciding  among 
the  cars  A,  B,  and  C.  The  difficulty  seems  to  be  in  focusing  on  the  factors 
which  discriminate  the  specific  options,  and  ignoring  factors  which  in  gen¬ 
eral  are  important  but  not  discriminating  in  this  decision  problem. 

If,  as  in  many  cases,  the  factors  are  all  diagnostic,  then  the  decision¬ 
maker  is  faced  with  assessing  the  relative  diagnosticity  or  relative  weight 
of  the  factors.  Procedures  for  assigning  relative  weights,  based  on  the 
diagnosticity  of  the  factors,  can  vary  from  a  simple  rank  ordering  to  more 
sophisticated  assignments  of  normalized  subjective  probabilities.  In  most 
cases,  however,  numerical  values  are  assigned  which  correspond  to  the  per¬ 
ceived  relative  weights.  While  there  is  no  technique  that  can  guarantee  to 
place  the  factors  in  optimal  order,  psychological  INTEGRATION  aids  are  de¬ 
signed  to  help  a  decisionmaker  determine  which  factors  are  diagnostic,  then 
assess  the  relative  importance  of  those  factors. 

Judgmental  and  Cognitive  Biases.  Psychological  research  has  documented 
several  judgmental  biases  which  can  severely  degrade  the  quality  of  decisions. 
It  is  very  important  to  recognize  that  these  judgmental  biases  are  not  inten¬ 
tional  and  are  so  pervasive  and  compelling  that  training  and  instruction  de¬ 
signed  to  help  decisionmakers  overcome  specific  biases  have  largely  been  in¬ 
effective.  Because  training  has  failed,  it  is  even  more  critical  that 
decision  aids  be  designed  to  compensate  or  help  minimize  the  impact  of  these 
judgmentally  degrading  biases.  While  it  is  beyond  the  scope  of  the  present 
paper  to  discuss  them  all  at  length,  a  few  will  be  briefly  outlined  to  demon¬ 
strate  the  need  to  address  such  factors  when  designing  decision  aids. 

One  biasing  factor  is  the  illusory  correlations  which  are  inferred 
about  the  relationship  of  two  events  A  and  B.  Because  A  and  B  are  observed 
to  co-occur  a  number  of  times,  they  are  believed  to  be  highly  correlated 
w. th  each  other.  However,  simple  co-occurrence  is  not  sufficient  evidence 


to  infer  correlation.  It  is  necessary  to  consider  how  often  B  occurs  and  A 
does  not.  For  example,  if  high  gas  mileage  cars  are  also  high  in  price,  a 
positive  correlation  might  be  assumed.  What  has  not  been  considered  is  the 
number  of  high  gas  mileage  cars  with  low  price  and  the  low  gas  mileage  cars 
with  high  price.  Decisionmakers  apparently  can  readily  search  for  positive 
instances  of  the  relationship  between  two  or  more  variables,  but  have  dif¬ 
ficulty  in  either  searching  for  or  assimilating  negative  instances. 

Decisionmakers  also  have  misconceptions  concerning  probabilistic  infor¬ 
mation.  The  probability  of  rolling  an  even  number  using  a  fair  die  is  .5; 
the  gambler's  fallacy  is  the  misperception  that  if  a  number  of  rolls  occurs 
without  an  even  number  that  the  likelihood  of  one  occurring  soon  is  increased. 
It  is  commonly  said  that  the  event  in  question  "is  due."  The  decisionmaker 
fails  to  consider  that  the  probabilities  are  based  on  thousands  of  rolls  and 
a  series  of  10  rolls  will  not  necessarily  exhibit  the  same  proportion  of  odd 
and  even  rolls.  It  is  illogical  to  expect  that  an  observed  low  or  high  fre¬ 
quency  will  be  compensatory  within  a  small  sample. 

Decisionmakers  also  have  a  t  idency  to  inaccurately  recall  the  confi¬ 
dence  with  which  they  made  a  decision  after  the  decision  outcome  is  known. 

In  such  hindsight  biases,  it  appears  that  once  the  outcome  is  learned,  de¬ 
cisionmakers  unintentionally  distort  their  perception  of  their  predecision 
processes.  A  related  problem  is  the  feeling  that  when  presented  with  new 
information,  the  decisionmaker  sees  the  information  as  obvious,  incorporates 
it,  then  fails  to  recognize  that  the  information  was  not  known  previously. 

Both  of  these  tendencies  have  been  dubbed  the  "I  knew  it  all  along"  effect. 

The  danger  in  such  a  bias  is  that  what  a  decisionmaker  can  learn  from  the 
decision  outcome  or  the  new  information  is  severely  inhibited. 

When  aids  have  been  constructed  to  help  structure  and  weigh  informa¬ 
tion,  as  well  as  minimize  judgmental  biases,  cognitive  limitations  will 
also  be  largely  overcome.  Simply  by  providing  an  explicit  structure  and 
making  weighted  information  and  necessary  calculations  readily  retrievable 
by  the  decisionmaker-,  limitations  such  as  memory  capacity  can  easily  be 
avoided.  Thus,  a  decisionmaker  is  relieved  of  the  burden  of  remembering 
all  the  information  and  evaluations  of  that  information.  In  addition  memory 
biases  such  as  the  disproportionate  ease  in  recalling  first  and  last  infor¬ 
mation  obtained  will  also  be  avoided. 

Example;  Integration  Aid.  A  very  critical  decision  problem  for  an 
Army  tactical  commander  is  to  select  the  most  advantageous  course  of  action 
to  pursue  in  attacking  enemy  forces,  terrain,  or  cities.  That  is,  what 
route  will  most  quickly  allow  the  commander's  forces  to  reach  their  desti¬ 
nation  and  accomplish  their  mission  with  the  fewest  forces,  least  equipment, 
and  fewest  losses.  Such  a  decision  has  obviously  grave  consequences  for 
many  people  as  well  as  the  commander.  The  problem  is  complicated  by  not 
only  these  consequences  but  by  the  extreme  time  pressure  under  which  the 
information  is  gathered,  evaluated,  integrated,  and  a  decision  made.  A 
computerized  (IBM  5100)  INTEGRATION  aid  has  been  developed  by  the  U.S.  Army 
Research  Institute  (Kibler  et  al.,  1978)  to  help  a  commander  both  focus  at¬ 
tention  on  and  critically  evaluate  relevant  information.  The  aid  has  been 
named  TACVAL  since  its  function  is  to  aid  in  tactical  evaluation  of  alter¬ 
native  courses  of  action. 
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The  structure  of  TACVAL  is  based  on  multiattribute  utility  theory, 
which  requires  that  the  decision  problem  be  broken  down  into  its  contribut¬ 
ing  component  parts  or  factors.  Thus,  the  first  step  in  designing  the  aid 
was  to  identify  and  define  an  exclusive,  but  not  exhaustive,  set  of  factors 
which  would  encompass  the  most  relevant  information  for  selecting  courses 
of  action  under  most  circumstances .  There  are  a  total  of  24  factors  grouped 
into  5  categories  organized  into  a  two-level  hierarchical  structure. 

To  use  TACVAL,  a  commander,  or  more  realistically  a  member  of  the  com¬ 
mander's  staff,  first  defines  the  courses  of  action  to  be  evaluated.  Then, 
each  course  of  action  is  independently  evaluated  on  each  of  the  24  factors. 
Such  evaluations  involve  assigning  a  numerical  score  to  both  the  value  of 
each  alternative  and  the  relative  weight  or  diagnostici.ty  of  each  factor 
for  making  this  particular  decision.  After  all  factors  are  evaluated  vari¬ 
ous  calculations  are  then  performed  based  on  the  user's  inputs  and  prepro¬ 
grammed  algorithms.  Two  of  these  calculations  are  a  normalized  weighted 
average  score  for  each  course  of  action  indicating  the  user's  relative  pref¬ 
erences  among  the  alternatives  and  a  sensitivity  analysis  which  identifies 
the  factors  which  are  most  sensitive  to  changes  in  the  user's  evaluation. 

In  addition,  there  are  options  for  editing  the  evaluations  based  on  new  in¬ 
formation  or  reevaluation  of  old  information  and  listing  the  relative  impor¬ 
tance  of  the  24  factors. 

In  terms  of  the  functions  of  INTEGRATION  aids  listed  in  Figure  1, 

TACVAL  performs  at  least  the  following:  (a)  The  user  searches,  requests, 
and  organizes  the  available  information  based  on  the  2-level  hierarchical, 
24-factor  structure.  Thus  all  the  relevant  information  is  systematically 
arranged  and  attention  to  irrelevant  information  can  be  minimized.  (b)  There 
are  explicit  instructions  and  scoring  conventions  built  into  the  aid  which 
help  the  user  in  a  step-by-step  fashion  to  evaluate  the  factors  and  assign 
numerical  weights.  Problems  in  determining  factor  diagnosticity  are  reduced 
by  explicit  elicitation  procedures.  (c)  Judgmental  biases  are  also  minimized 
by  providing  the  user  with  a  hard  copy  of  all  the  factors  and  evaluations 
of  the  alternatives  on  those  factors.  Hindsight  biases  are  eliminated  since 
the  user  has  a  complete  record  of  all  evaluations,  updating,  reevaluations, 
and  resulting  calculations.  (d)  Cognitive  limitations  such  as  memory  over- 
oad  are  minimized  since  all  evaluations  and  factors  are  explicit  and  inf or¬ 
ation  is  organized  within  the  framework.  In  addition  since  each  factor  is 
evaluated  independently,  the  user  need  only  be  concerned  with  information 
relevant  to  that  factor  and  need  not  be  concerned  with  recalling  other  fac¬ 
tors  or  information.  Finally,  psychological  problems  in  combining  the  evalu¬ 
ations  and  weight  of  the  factors  are  eliminated  since  they  are  arithmetically 
computed . 


EVALUATION 

The  purpose  of  the  EVALUATION  component  of  the  Decision  Support  Frame¬ 
work  is  to  provide  critical  feedback  for  the  decision  aid  designer  concerning 
the  success  of  the  aids.  In  this  context  it  is  not  the  correctness  of  the 
DECISION  that  is  assessed,  but  the  value  of  the  decision  aids  in  making  a 
rational  decision.  As  shown  in  Figure  1,  the  EVALUATION  step  serves  at 
least  three  factions,  each  of  which  will  be  briefly  discussed. 
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c. . 


Validity  and  Reliability 


If  the  factors  selected  do  not  allow  critical  information  to  be  in¬ 
cluded  in  assessments  or  the  input  judgments  do  not  accurately  reflect  the 
feelings  of  the  decisionmaker ,  then  the  output  of  the  aid  will  be  mislead - 
ing.  Validity  assessments  should  include  extensive  consultation  with  ex¬ 
perts  to  assure  the  factors,  their  labels,  and  definitions  do  in  fact  accu¬ 
rately  capture  the  scope  of  the  decision  problem.  In  addition  the  judgments 
on  those  factors  must  reflect  the  true  opinions  of  the  user.  While  evalu¬ 
ation  of  the  validity  of  input  judgments  can  be  elusive,  checks  such  as 
internal  consistency  of  judgments  and  questioning  of  the  user  should  be  at¬ 
tempted.  Measures  of  the  reliability  of  the  user's  judgments  are  also  neces¬ 
sary,  not  so  much  to  assess  the  user,  but  to  be  sure  the  procedure  unam¬ 
biguously  elicits  the  user's  responses.  Experimental  validcition  of  both 
factors  and  judgments  could  include  comparisons  across  decision  problems 
and  decisionmakers  as  well  as  variations  in  the  factors  and  information. 

A  related  validation  issue  is  the  credibility  of  the  decision  aid 
structure,  process,  and  output  to  the  decisionmaker.  While  the  decision¬ 
maker  may  not  be  the  best  expert  on  psychological  biasing  factors  or  infor¬ 
mation  organization,  the  ultimate  fidelity  of  the  aid  depends  on  the  decision¬ 
maker's  acceptance.  Thus,  efforts  should  be  made  to  assess  the  user's 
beliefs  and  confidence  in  the  aid  as  well  as  abilities  to  make  valid  judg¬ 
ments  of  the  information. 


Flexibility 

Since  most  decision  situations  are  not  completely  static,  a  decision 
aid  can  never  include  all  factors  which  may  be  of  conceivable  importance  in 
unforeseen  circumstances.  Therefore,  it  is  necessary  to  assess  the  flexi¬ 
bility  of  the  decision  aid  to  incorporate  unique  information,  factors,  or 
events.  One  possible  approach  to  evaluate  flexibility  is  to  present  a  user 
with  unexpected  but  obviously  important  information,  then  assessments  of 
how  and  to  what  degree  the  information  is  incorporated  into  the  decision 
aid  structure  can  then  be  made. 


Improved  Decisions 

While  there  is  agreement  from  many  disciplines  that  aids  are  necessary 
to  support  complex  and  consequential  decisionmaking,  there  is  very  little 
reported  evidence  that  decision  aids  do  in  fact  lead  to  improved  decision. 
Perhaps  the  most  basic  reason  for  this  failure  to  evaluate  decision  aids  is 
the  lack  of  a  criterion  for  assessing  the  quality  of  dicisions.  In  some 
cases  the  actual  consequences  are  too  far  removed  rrom  the  decisions,  while 
in  others  there  is  no  realistic  context  in  which  to  test  the  decision  aids, 
e.g.,  a  tactical  commander  at  war.  However,  in  the  bulk  of  the  cases,  an 
objective  criterion  could  certainly  be  developed  even  if  only  in  hypotheti¬ 
cal  or  contrived  contexts.  The  crux  of  the  evaluation  issue  lies  in  opera¬ 
tionally  defining  the  DECISION  and  ^EQUI'.iEMENTS  before  the  aids  are  even 
developed.  Without  clear  objective  definitions  of  the  purposes,  goals, 
and  the  decision  itself,  meaningful  measiures  of  the  contribution  of  the 
decision  aids  to  a  rational  decision  cannot  be  made.  Assuming  operational 
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measures  have  been  developed,  experimental  comparisons  of  decisions  uicde 
with  aids,  without  aids,  with  various  modifications,  euc.,  should  be  made 
tc  assess  the  contribution  of  the  aids  to  making  a  rational  decision. 

As  a  final  point  on  evaluation,  it  should  be  noted  that  in  some  situ¬ 
ations  the  evaluation  should  include  me.  jures  other  than  the  rational  de¬ 
cision.  Based  on  the  analysis  of  the  decision  REQUIREMENTS,  perhaps  mea¬ 
sures  such  as  the  type  and  amount  of  information  used,  or  the  degree  to 
which  different  decisionmakers  resolve  conflict  about  the  decision,  etc., 
may  also  be  valid  evaluation  criteria.  At  times  a  decisionmaker  is  just 
as  concerned  with  how  the  decision  is  made  as  with  the  decision  itself. 


CONCLUSIONS 

The  Decision  Support  Fr?.*uework  categorizes  and  relates  different  types 
of  decision  aids  based  on  the  functions  they  perform  for  a  decisionmaker. 
However,  beyond  its  use  as  an  organizational  tool,  it  requires  the  decision 
aid  designer  to  carefully  analyze  the  decision  to  determine  the  requirements 
for  aiding  the  various  functions  which  must  be  performed  by  the  decision¬ 
maker,  as  well  as  operationally  define  the  decision  criteria  for  valid 
evaluation  of  the  decision  aid.  In  short,  using  a  complete  framework  such 
as  presented  here  serves  as  a  guide  to  help  decision  aiders  develop  logical 
and  useable  decision  aids. 
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